Metal additive manufacturing (AM) has attracted much attention in recent years due to its ability of producing parts with complex geometry. Unfortunately, the large temperature gradient during fabrication leads to residual stresses which undesirably result in distortion and even crack of as-built parts. A computational framework is used to study how residual stresses form and evolve in AM parts at the length scale of individual grains, including a multi-physics thermal-fluid flow model, a phase field model for grain growth and a crystal plasticity finite element model. First, this framework is validated by comparing the lattice strain with experimental results in different grain families in two samples made of 316L stainless steel, which were produced by laser powder-bed-fusion with two different sets of process parameters. The relationship between residual stress, plastic strain and grain orientation near the top surface of the samples is then investigated. The residual stresses are observed to form during laser scanning due to compression followed by tension around the molten pool, thus the shape of the molten pool has a significant influence on the residual stress distribution. Redistribution of the plastic deformation during cooling stage is predicted and the residual stresses with greater magnitude occur along the laser scanning direction. This work provides useful insight into the mechanism of microscale residual stress generation and evolution in AM parts.